Modeling the brain has been a long-standing goal of scientists and engineers alike. Many neural networks and neurally-inspired algorithms have been proposed, designed, and built with different goals and purposes in mind. Some of these algorithms have been developed to further the understanding of the human brain through detailed models. Others have developed these algorithms to mimic the decision-making and data classification abilities of the human brain. Computational hardware designers have even captured the ultra-low power processing capability of the brain, which is inherently more energy efficient than typical microprocessors.
Regardless of the neural algorithm, its implementation, or its purpose, there is one commonality across all cases: these algorithms model neurons and the connectivity between them. In the human brain, a neuron may connect to tens or hundreds of thousands of other neurons. In biological neural networks, supporting this high-scale connectivity isn't much of an issue, since the dendritic and axonal branches of neurons can grow in 3-dimensional space to form direct connections with other neurons. However, with artificial neural software models and neuromorphic hardware implementations, this high-scale connectivity presents a significant challenge.
Neuromorphic hardware designs such as the IBM Neurosynaptic Core have used a crossbar, which allows for all-to-all connectivity between 256 neurons; however this solution clearly fails to scale to the connectivity levels of biological neurons. Furthermore, the crossbar structure only supports connectivity with the same population of 256 neurons. In biological systems, connectivity occurs across populations and with much higher degrees of fan-in and fan-out than supported by this hardware crossbar.
Conventional approaches for scaling hardware connectivity to larger sets of neurons take their inspiration from the field of interconnection networks, where all-to-all approaches, i.e. crossbars, are known to not scale beyond a few tens of endpoints. Instead, direct (where every node is an endpoint) and indirect (where some nodes are just purely for routing) network topologies are used to connect large numbers of endpoints together by enabling forwarding or routing of messages via multiple hops in the network. These schemes are relatively efficient to implement and may be scalable to many endpoints, but still do not provide robust scaling to anywhere near biological scale.
Computer systems scale to millions of nodes by employing a combination of hardware and software, as demonstrated in the architecture of the Internet. Here, the interconnection network relies on hierarchy and locality of communication patterns to enable systems with millions of nodes. It does so by providing robust connectivity to localized groups, growing in hierarchical clusters to larger aggregate node counts, but providing only limited connectivity and bandwidth to non-local nodes. The Internet's hierarchical connectivity has an analogue in neural systems, where there are far more local connections than global (distal) connections, so hierarchical approaches seem like an appealing solution.
Hybrid hardware/software as well as pure software implementations for scalable neural connectivity enable networks with far more connections than direct hardware solutions like IBM's Neurosynaptic crossbar. The number of connections need not be supported by physical connections; rather, connections between neurons can be stored in memory, allowing for much more universal neural connectivity schemes and architectures. However, this flexibility comes at a price: as the system is scaled to biological connectivity levels, the amount of memory needed to store the connectivity graph grows drastically. Each endpoint must be assigned an address, and every endpoint has to maintain a list of its connections, ideally for both upstream and downstream neurons.
Consider as an example a billion neuron network with 10,000 connections projecting from each neuron: this would require log2(109) bits/address×109 nodes×(24 addresses/node)=75 TB of storage just for the connections. Even a modest million-neuron network would require 50 GB to store the connectivity graph. The sheer amount of memory needed, the power consumed by that memory, as well as the latency to access it, quickly become the bottleneck in such systems.
Available solutions for implementing connectivity in neural algorithms suffer from limited connectivity, poor flexibility, high overheads and memory requirements, or all of the above. While these issues are most evident when considering the connectivity of large scale neural algorithms, the same problems exist for many of the other neuron parameters that are modeled in either software or hardware systems, such as the neuron's firing threshold, the synaptic weights, or the neuron's membrane leak. In traditional techniques, each of these, and many more, parameters must be stored in memory and accessed often during the execution of the neural algorithm. Therefore, in both hardware and software implementations, there is a clear need for a more scalable and efficient neural system.